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ABSTRACT 

The use of computer-based automated assignment systems in economics has expanded significantly in 
recent years. The most widely used system is Aplia which was developed by Paul Romer in 2000. Aplia 
is a computer application designed to replace traditional paper-based assignments in economics. 

The main features of Aplia are: (1) interactive content including problem sets, experiments and 
news analysis; (2) digital editions of a textbook; (3) assignment sets that are customised to specific 
textbooks; and (4) immediate feedback for both students and instructors. Its ability to present the 
dynamics of diagrams and graphs is critical to its use in economics. 


This paper analyses the effectiveness of Aplia and traditional paper-based assignments and tutorials 
using summative assessment results. The analysis is based on a managerial economics course that 
was taught to over 380 students at NUI Galway in the first semester of 2008-09. The course was 
designed so that each student was required to complete eight assignments for 25% of the total 
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marks available for the course. They completed six of the eight assignments by Aplia and the 
remaining two by paper. 


The final exam was organised into eight sections with each section corresponding to a 
particular assignment. Our basic test is to examine whether a student's performance in a 
particular section of the exam is affected by whether the student completed the corresponding 
assignment on paper or online. We also examined if how the student performed on a particular 
assignment, regardless of the type, predicted how well they did on the corresponding 
examination question. We found little statistical evidence in support of either hypothesis. 

INTRODUCTION 

Economists like to point out that everybody faces trade-offs. A country can spend more on 
health but will then have less to spend on new cars. An individual can choose to work longer 
hours but will then have less time to spend with her family. And so it goes with large classes 
and continuous assessment. A lecturer can assign regular graded assignments to students but 
will then have little time to do all the other things that a lecturer is supposed to do. Have a 
look at the first or second chapter of any textbook in introductory economics and you will find 
a precise definition of opportunity cost as well as a concept called the production possibilities 
frontier which neatly illustrates the essential concepts of scarcity and choice (Mankiw and 
Taylor, 2006; Turley et at). 

Dismal as it may be, economics also teaches us that trade-offs can change, often dramatically 
so, over time. Individuals and societies can produce far more goods and services today than 
they could fifty or a hundred years ago. The reason generally lies in technological change 
which allows the production possibilities frontier to be pushed out further and further from the 
origin. The issue we are concerned about in this paper is whether technological change could 
change the tradeoffs facing lecturers who wish to give regular assignments to large classes. 1 

Economics has long been taught to large classes in first and second year at National University 
of Ireland (NUI) Galway. Brendan Kennelly has taught either principles of economics (first 
year) or managerial economics (second year) at NUI Galway since 1998. Like most economics 
lecturers, he believed that economics is best learned by doing problems and assignments 
rather than by learning off theories and definitions but felt unable to give regular assignments 
because they could not be graded. For many years the solution was to give the students a 
certain number of assignments each semester and to grade only one of them. To ensure that 
the students had an incentive to do all of the assignments the assignment that was graded 
was not returned to the students until the semester was finished. But this meant that one 
of the main goals of assignments - to give students feedback on what they knew and (more 
importantly) what they did not know - could not be realised. It was a decidedly second best 
solution. 

While Brendan was struggling with large classes in Galway, an economist called Paul Romer 
was developing an online automatically graded assignment service called Aplia for his 
economics students at Stanford University. While it was initially designed simply as a product 
for his own classes Romer quickly realised that other economists might be interested in his 
product and its use has expanded dramatically in the last ten years. 2 Aplia is now used in well 
over 1,000 universities around the world. Over 580 million answers have been submitted and 

1 By large classes we mean classes with at least 150 students. Aplia can be used with classes of any size. Our focus on large classes is driven by the fact that 
managerial economics is taught to large classes at NUI Galway. 

2 In some respects Paul Romer was an unlikely candidate to develop a system of online assignments in economics. He is best known as an economic theorist and his 
work on what is called new growth theory is widely respected by the profession and will very likely result in him being awarded the Nobel Prize in economics. 
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automatically graded and every minute about 200 more answers are submitted. 


The main features of Aplia 3 are: (1) its interactive content including problem sets, experiments and 
news analysis; (2) the digital editions of a textbook; (3) the assignment sets that are customized 
to specific textbooks; and (4) the immediate feedback for both students and instructors. It provides 
facilities for both formative and summative assessment. Aplia appeals to those with responsibility for 
educational budgets on the grounds of cost. 4 It also appeals to educationalists who value the direct 
student interaction with the formative assessment plus immediate feedback on both summative and 
formative assessment. 


In 2006, Brendan Kennelly and his colleague David Duffy became the first economists in Europe to 
use Aplia. Our aim in this research project is to evaluate the effectiveness of Aplia, by comparing 
Aplia and traditional paper-based assignments using summative assessment results. Do students learn 
as much from Aplia as they do from traditional paper-based assignments? We also wanted to find out 
how students used Aplia and what they thought of it as a learning tool for examination preparation 
purposes. 

DATA AND ANALYSIS 

The research is based on a managerial economics course that was taught to over 380 students at 
NUI Galway in the first semester of 2008-09. Previous work in this area has yielded mixed results. 
Nguyen and Trimarchi (2010) found that Aplia, and a similar product, MyEconLab were responsible 
for increasing the average class mark by about 2% regardless of whether the technology used was a 
required or optional course component. O'Dea and Ring (2008) found that the percentage of Aplia 
questions attempted had a significant negative impact on test performance. They also found that 
the percentage of Aplia questions answered correctly had no effect on test performance. These and 
other studies on Aplia have compared the performance of one group of students who did assignments 
on Aplia with another group of students who did assignments on paper (Lee et a l, 2010). A unique 
feature of our analysis is that all of the students were required to do six of their assignments 
on Aplia and two on paper and had no choice over which assignments to do on paper. The final 
exam was organised into eight sections with the material examined in each section corresponding 
to a particular assignment. Our basic question is to examine whether a student's performance 
in a particular section of the exam is affected (a) by how s/he performed in the corresponding 
assignment; and (b) by whether the student completed the corresponding assignment on paper or 
online. We have up to six observations on exam performance for each student with corresponding 
data on assignment performance. 


A panel data set is one where the variable being analysed varies over two dimensions. The variation 
across both students and sections in exam performance meant that our data could be analysed using 
panel data techniques. Standard statistical tests indicated that a fixed effects model was appropriate. 
The dependent variable was the student's performance in each section of the exam and the critical 
explanatory variables were the student's score in the corresponding assignment and a dummy variable 
that indicated whether the student had done the assignment on Aplia or on paper. The coefficients 
on the assignment score and the Aplia/paper dummy were both statistically insignificant (i.e. their 
p-values are too high). In other words we do not find any evidence that one's performance in an 
assignment or whether the assignment is done on Aplia or on paper has any effect on how one does 
in the corresponding section of the exam. We also included a dummy variable for each section to 
capture common differences across sections. These dummies were all significant. Table 1 contains the 
detailed econometric results. 


3 For a demonstration about how Aplia works, see www.aplia.com. Aplia provides services in disciplines such as statistics, accounting and psychology as well as economics. 

4 Aplia offers various pricing options to professors and students. Some institutions have reduced the number of face-to-face recitations after adopting Aplia. Depending on what 
pricing options are available the total cost of buying the textbook and the Aplia service may be higher or lower for students than just buying the textbook. 
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Table 1: Fixed effects model on section-specific exam performance with Aplia/Paper 
dummy 


Constant 

22.28 

(0) 

Assignment Mark 

.003 

(.918) 

Aplia/paper dummy 

-.55 

(.35) 

Dummy for Section 2 

2.8 

(.001) 

Dummy for Section 3 

-2.0 

(.04) 

Dummy for Section 4 

3.22 

(.001) 

Dummy for Section 5 

- 7.25 
(0) 

Dummy for Section 6 

-4.11 

(0) 

Dummy for Section 7 

-5.01 

(0) 

Dummy for Section 8 

-3.1 

(.001) 

Observations 

1,351 

Sigma_u 

7.61 

Sigma_e 

8.74 

rho 

0.43 


p-values in parentheses 

Note: Dependent variable is the score that a student received in each section of the exam 

Note: Aplia/paper dummy is specified with 1 = assignment completed on aplia and 0 = 
assignment completed on paper. 

Our analysis indicated that completion rates on the Aplia assignments were higher than on the 
paper assignments. In a formal statistical model we found that the Aplia/paper dummy had a 
positive and significant effect on the likelihood that a student completed an assignment in 
this model. However we did not find any evidence that merely completing an assignment had a 
positive effect on one's performance in a particular section of the exam 5 . 

In the final week of the semester the students were surveyed about their experience with 
Aplia and the paper assignments. A very substantial majority of students believed that Aplia 
assignments had a positive effect on their overall understanding of the topics covered in the 
course and a majority of students believed that they learned more from the Aplia assignments 
than the paper ones. A majority of students also said that they preferred Aplia to paper 
assignments. When asked why, only a quarter said they did so because they thought they 
learned more from Aplia. A much larger percentage said that they preferred Aplia because they 
found the Aplia assignments easier to complete. 


5 


Detailed results are available in Kennelly et al (2010) which contains several more tables of results as well as more detail on the implications of our results. 
This paper is available on request. 


Over 85% of the students said that they would like to use Aplia in other courses. When asked why, 
the students said they liked having a substantial proportion of the courses awarded for continuous 
assessment. They also liked the practice assignments on Aplia and said that Aplia was of particular 
benefit with regards to the use of graphs in economics. This agrees well with an earlier survey by 
Kennelly and Duffy (2007). 

We supplemented the survey with two focus group meetings, one held during the semester and the 
other held a few months after the final exam. One topic we explored in the second meeting was 
whether the students regarded Aplia as a good tool to prepare them for examination questions. The 
students regarded Aplia as more of a complement to a traditional tutorial system than a substitute 6 . 

We learned at the first focus group meeting that some of the students regarded the Aplia assignments 
as puzzles that could be figured out with the aid of textbook or practice assignments rather than 
pedagogically useful learning devices. 

CONCLUSION 

Our statistical analysis revealed little evidence that online assignments were not as good as paper- 
based assignments in helping students learn key concepts and techniques in a second year economics 
course. Given that online assignments take up much less of a lecturer's time and given their 
popularity among students, our evidence indicates that Aplia is a technological improvement that is 
valuable and one that should be seriously considered by lecturers around the world. 


On the other hand, the absence of a positive correlation between performance in assignments and 
performance in exam section may be regarded as disappointing. One issue that we think needs more 
consideration is the link between what the students learn in assignments and the kind of questions 
that students face in exams. This is a key issue for Aplia as the technology restricts the kind of 
questions that can be asked in the assignments. The Aplia questions tend to be modelled very closely 
on examples that are worked out in the textbook and students are often able to answer the questions 
by matching the questions with the corresponding examples in the book. 


A unique feature of our paper is that we have up to six observations on exam performance for each 
student with corresponding data on assignment performance. However this may not have been as 
important as we thought when we devised the study. It may be the case that students think in a 
compartmentalised way about assignments and exams (some of the comments in the focus group 
meetings suggested this). A closer link between assignments and exam performance might be found 
if there had been one or two midterm examinations in the course with a final exam that focused on 
the material that had been taught in the final part of the course. Whether that link would be stronger 
for paper or online assignments remains an open question. 
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